Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support for HTTP transport #13

Merged
merged 7 commits into from
Apr 5, 2024
Merged

feat: support for HTTP transport #13

merged 7 commits into from
Apr 5, 2024

Conversation

jerrinot
Copy link
Collaborator

@jerrinot jerrinot commented Apr 4, 2024

This pull request introduces support for HTTP transport.

QuestDB ILP client supports HTTP transport in versions 7.4.0+ thus the client dependency version is updated too.

Configuration
The connector now supports client configuration strings as documented in the client documentation. Set the client configuration string under the key client.conf.string.

Example:

{
  "name": "my-connector",
  "config": {
    "tasks.max": "4",
    "connector.class": "io.questdb.kafka.QuestDBSinkConnector",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "topics": "mytopic",
    "value.converter.schemas.enable": "false",
    "timestamp.field.name": "ts",
    "timestamp.units": "nanos",
    "client.conf.string": "http::addr=questdb:9000;auto_flush_rows=10000;auto_flush_interval=1200000;retry_timeout=60000;"
  }
}

The old client configuration keys (host, username, token, tls, etc) are considered deprecated and should not be used. They cannot be used together with the client configuration string.

Kafka-specific configuration options (timestamp.units, topics, etc.) are not part of the client configuration string and still have their own keys. See the example above.

Advantages of the HTTP transport
The HTTP transport implements proper error reporting, automatic retries for recoverable errors and simpler configuration. When combined with QuestDB server deduplications it allows Exactly-Once semantics.

Known Issues
Due to a bug in QuestDB client 7.4.0 it's recommended to disable interval-based flushes. See questdb/questdb#4372 for details. This will be resolved in the next version.

@jerrinot jerrinot marked this pull request as ready for review April 5, 2024 10:14
@jerrinot jerrinot merged commit e3792ce into main Apr 5, 2024
2 checks passed
@jerrinot jerrinot deleted the jh_http branch April 5, 2024 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant